Mml, Hybrid Bayesian Network Graphical Models, Statistical Consistency, Invariance and Uniqueness

The problem of statistical — or inductive — inference pervades a large number of human activities and a large number of (human and non-human) actions requiring ‘intelligence’. Human and other ‘intelligent’ activity often entails making inductive inferences, remembering and recording observations from which one can make inductive inferences, learning (or being taught) the inductive inferences of others, and acting upon these inductive inferences. The Minimum Message Length (MML) approach to machine learning (within artificial intelligence) and statistical (or inductive) inference gives us a trade-off between simplicity of hypothesis (H) and goodness of fit to the data (D) [Wallace and Boulton, 1968, p. 185, sec 2; Boulton and Wallace, 1969; 1970, p. 64, col 1; Boulton, 1970; Boulton and Wallace, 1973b, sec. 1, col. 1; 1973c; 1975, sec 1 col 1; Wallace and Boulton, 1975, sec. 3; Boulton, 1975; Wallace and Georgeff, 1983; Wallace and Freeman, 1987; Wallace and Dowe, 1999a; Wallace, 2005; Comley and Dowe, 2005, secs. 11.1 and 11.4.1; Dowe, 2008a, sec 0.2.4, p. 535, col. 1 and elsewhere]. There are several different and intuitively appealing ways of thinking of MML. One such way is to note that files with structure compress (if our file compression program is able to find said structure) and that files without structure don’t compress. The more structure (that the compression program can find), the more the file will compress. Another, second, way to think of MML is in terms of Bayesian probability, where Pr(H) is the prior probability of a hypothesis, Pr(D|H) is the (statistical) likelihood of the data D given hypothesis H, − logPr(D|H) is the (negative) log-likelihood, Pr(H|D) is the posterior probability of H given D, and Pr(D) is the marginal probability of D — i.e., the probability that D will be generated (regardless of whatever the hypothesis might have been). Applying Bayes’s theorem twice, with or without the help of a Venn diagram, we have Pr(H|D) = Pr(H&D)/Pr(D) = (1/Pr(D)) Pr(H)Pr(D|H). Choosing the most probable hypothesis (a posteriori) is choosingH so as to maximise Pr(H|D). Given that Pr(D) and 1/Pr(D) are independent of the choice of